Augmented reality system for conveyor system and method

ABSTRACT

An augmented reality system includes a display device defining a display area, a database containing a three-dimensional model of a conveyor system, a communication link between the database and the display device, and a processor. The processor is operable to display the three-dimensional model of the conveyor system on the display area. The display allows a human user to see a real time combined view of the real physical environment, including a building and features thereof, and the three-dimensional model of the factory automation system within the building. A user interface has controls operable to effect one or both of: movement of the displayed conveyor system within the building, and measurement between features of the building and the displayed conveyor system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/532,555, filed Jul. 14, 2017, the entire contents of which areincorporated by reference herein.

BACKGROUND

Virtual reality and augmented reality equipment has become increasinglyavailable and affordable. However, the majority of the systems aredesigned for and operated primarily to provide entertainment. Therefore,opportunities exist for using the tools of these technologies in waysthat provide industrial utility

SUMMARY

In one aspect, the invention provides an augmented reality systemincludes a display device defining a display area, a database containinga three-dimensional model of a conveyor system, a communication linkbetween the database and the display device, and a processor. Theprocessor is operable to display the three-dimensional model of theconveyor system on the display area. The display allows a human user tosee a real time combined view of the real physical environment,including a building and features thereof, and the three-dimensionalmodel of the factory automation system within the building. A userinterface has controls operable to effect one or both of: movement ofthe displayed conveyor system within the building, and measurementbetween features of the building and the displayed conveyor system.

In another aspect, the invention provides a method of operating anaugmented reality system in a building. A display device is provided toa human user in the building, the display device defining a displayarea. A three-dimensional model of a conveyor system is displayed to thehuman user, while simultaneously allowing the human user to see realphysical features of the building in real time to give the impressionthat the conveyor system is installed in the building. Displaying thethree-dimensional model of the conveyor system includes presenting aconveyor system control panel. In response to a command from the uservia an input to a user interface, virtual operation of the displayedconveyor system is started.

In yet another aspect, the invention provides a method of operating anaugmented reality system in a building. A display device is provided toa human user in the building, the display device defining a displayarea. A three-dimensional model of a conveyor system is displayed to thehuman user, while simultaneously allowing the human user to see realphysical features of the building in real time to give the impressionthat the conveyor system is installed in the building. With a userinterface, a measurement between the three-dimensional model and atleast one utility feature of the building is performed and/or a movementof at least a portion of the displayed three-dimensional model to a newposition within the building is performed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view of an augmented reality system according toone aspect of the invention, displaying a first conveyor system modelfor interaction with the user.

FIGS. 2-5 illustrate a second conveyor system model displayed by theaugmented reality system, the second conveyor system being shown in avariety of operational states during virtual operation.

FIG. 6 illustrates a user interface including various applets for theuser to interact with the augmented reality system.

DETAILED DESCRIPTION

Before any embodiments of the invention are explained in detail, it isto be understood that the invention is not limited in its application tothe details of construction and the arrangement of components set forthin the following description or illustrated in the accompanyingdrawings. The invention is capable of other embodiments and of beingpracticed or of being carried out in various ways.

The present application relates to industrial conveyor system design andcorresponding construction or installation at a factory site, forexample an industrial building. In particular, the invention provides anaugmented reality system operable to display and enable selectiveinteraction with a virtual model of one or more conveyor systems for themanufacture, transit, painting, coating/finishing, thermal treatment,etc. of parts, which may include any type of industrial work piece. Suchconveyor systems may provide for lateral transport, lifting/lowering,and/or changing the inclination of conveyed parts throughout one or morework stations of a factory. Work stations can include forming stations,cutting stations, welding stations, cleaning stations, spray booths,ovens, dip coating tanks and other known types of work stations utilizedfor the factory to convert raw materials or other starting work productsto finished or semi-finished work products. Non-limiting examples ofconveyor systems and work stations can be found throughout the followingpatents of SST Systems, Inc.: U.S. Pat. Nos. 9,468,944, 7,712,662,8,066,185, 8,376,230, 8,701,993, 8,505,712, 9,016,464, and 9,409,730,although the invention is not necessarily limited to the types ofconveyors and work stations disclosed therein. Aspects of the presentinvention can also be applied to other types of systems, including (byway of example only) water and wastewater treatment systems.

In one construction, an augmented reality (AR) system 100 includes amemory device, or “database”, 104 having electronic storage media forholding data representing a three-dimensional model of a conveyorsystem. The AR system 100 further includes a processor 108, atransceiver 112, and a display device 116. The processor 108 can operateto display the three-dimensional model of a conveyor system on thedisplay device 116 via the transceiver 112. For example, the transceiver112 can wirelessly communicate (e.g., via Bluetooth®, Wi-Fi™, etc.) totransmit electrical signals to the display device 116. In otherconstructions, the display device 116 may be physically coupled fortransmitting electrical signals rather than wirelessly. Some or all ofthe memory device 104, the processor 108, and the transceiver 112 can beincorporated in a personal computer and/or a server operating on anetwork. The display device 116 can be in the form of eyewear (e.g.,goggles, glasses, etc.) to be worn on the face of a human user. As such,the display device 116 can include a transparent or semi-transparentdisplay area that allows the user to see through to the real physicalsurroundings. Examples of such devices can include a variety ofso-called “smart glasses” or AR glasses such as Google Glass™, EpsonMoverio BT-100™, Microsoft HoloLens™, or Sony SmartEyeglass™ amongothers. The display device 116 can also include a forward-facing cameraor multiple cameras including at least one forward-facing camera tocollect optical data of the real physical environment in front of thehuman user when wearing the display device 116. In some constructions,this allows for a display area of the display device 116 to be partiallyor fully opaque, with an electronically generated display operable todisplay a combined view of the real physical environment from theoptical data along with the virtual three-dimensional model of theconveyor system. Such a system operates closer in principal to virtualreality (VR), however, the objective remains to allow the user tointeract with and view the real physical environment and not totransport the user out of the real physical environment. It should benoted that in some embodiments, one or more cameras (described above) ofthe AR system 100 can be in other locations on the user's body, such asany suitable body-mounted camera located in the user's head, shoulder,arm chest, and the like.

As an alternative to eyewear, the display device 116 can take the formof a tablet computer, a smart phone, or other personal electronicdevice. Similar to the description above, the display area, or screen,of such a device can be operated to show a combined view of the realphysical environment (e.g., produced by optical data from aforward-facing camera) along with the three-dimensional model of theconveyor system. When such a handheld device is used, it becomes aportal for viewing the three-dimensional model of the conveyor systemplaced into the real physical environment.

Whether the display device 116 takes the form of eyewear or a handheldelectronic device, the processor 108 is operable to display athree-dimensional image or “model” 124 of the conveyor system at apredetermined location within the real physical environment. As usedherein and in the appended claims, the term “model” is a shorthand wayto refer to the virtual conveyor system that is rendered to the user sothat the user sees the conveyor system as if it is physically present,allowing the user to move around, along, and optionally into the virtualconveyor system. Although the model can be a CAD model created in anymanner and having any file format, the term “model” is not limited toCAD models and related data, and can instead be or include other objectsand artifacts retrieved and displayed by the processor 108 upon thedisplay device 116, such as one or more images of real equipmentdisplayed upon the display device 116.

The location of display of the model 124 can be initially fixed to allowthe user to walk around, through, and/or into the conveyor system as ifit were present in the real physical environment, but the location maybe movable by the user as well. In the illustrated example of FIG. 1,the conveyor system of the three-dimensional model 124 includes aworkstation (e.g., an enclosed workstation such as an oven) 128, amovable doorway 132 that can be opened and closed, a control station 134and at least one conveyor track 136 supporting one or more work pieces140. Although the real physical environment of the model 124 may varywidely depending upon the building site, type of conveyor systeminstallation, etc., the environment can be an industrial building havinga floor 144, at least one wall 146 extending upwardly from the floor144, and a ceiling structure 148 consisting of ceiling panels, beams,rafters, etc. Although not shown, the real physical environment can alsoinclude other machines, work stations, equipment, or conveyor devicespresent at a particular industrial site. For example, the industrialsite may be in the process of initial installation or setup, orrebuilding, refurbishment, or repurposing. The industrial building caninclude any number of fixed utility features (e.g., drain 152A, a fluidsource 152B, an electrical source 152C, ventilation elements 152D, 152E)or other features such as a doorway, a storage facility, handlingequipment, a railing, a walkway, etc. In the illustrated construction,the floor 144 includes at least one drain 152A leading to a wastecollection system. Along the wall 146, the illustrated industrialbuilding includes a fluid source 152B (e.g., a line and/or junction forsupplying water, hydraulic fluid, or a gas) and an electrical source152C (e.g., electrical line and/or junction box). Further, the wall 146and/or the ceiling structure 148 can include one or more ventilationelements 152D, 152E such as fresh air inlets, air outlets, or airreturn/delivery duct openings of an air handling system, which mayoptionally include air treatment devices (e.g., filters or otherdecontamination devices).

When the model 124 of the conveyor system is displayed in the realphysical environment (e.g., building) in which it is to be installed,many advantageous features or operations become available to the user.Prior to or during early stages of any physical construction of theconveyor system at the real physical building location, display andinteraction with the model 124 allow the user to gain additionalinsights and understanding of the effects of physically installing theconveyor system according to the model 124 at a particular location inthe real physical environment. In addition to observing the model 124,the user may interact with the model 124 in any one or more of a varietyof ways, some of which are described in further detail below. It is alsonoted that observing or inspecting the conveyor system model 124 caninclude stepping inside a virtual interior of the conveyor system model124.

In order to interact with the model 124, the AR system 100 can include auser interface 160 as illustrated in FIG. 1. The user interface 160 canbe a physical handheld device having any one or more of: buttons, knobs,joysticks, dials, touch pads or other types of human machine interfaceelements. In constructions where the display device 116 is not eyewear,but rather a personal electronic device with a screen, the userinterface can form part of the image displayed on the screen. Inconstructions where the display device 116 is eyewear, the eyewear caninclude all or a portion of the user interface 160 by way of any one ormore human machine interface elements integrated into the display device116 (e.g., along the sides near the user's temples, above the eyes,etc.). In yet other constructions, some or all of the human machineinterface elements are not physical elements, but control elementsoperable to interface with the processor 108 through voice commandsand/or gestures from the user. The user interface 160 can be coupled incommunication with the processor through the transceiver 112 (e.g.,wirelessly), or the user interface 160 can be directly coupled to orintegrated with the processor 108. In some embodiments, the userinterface 160 is or includes a brain-computer interface enabling a userto perform one or more functions described herein. Such brain-computerinterfaces can include a helmet or headgear wearable by the user, and insome embodiments is integrated into or part of the display device 116also worn by the user.

In one aspect, a method of using the AR system 100 includes moving themodel 124 within the real physical environment. In a first step, themodel 124 is displayed at a predetermined location within the realphysical environment. The user can then view and/or inspect features orportions of the model 124 as displayed in the real physical environmentin the predetermined location. The user can then use the user interface160 to move the model in any one or more of the X-direction, theY-direction, and the Z-direction by inputting a movement command. Theprocessor 108 refreshes the model 124 to be re-presented at the newlydefined location. The user can then view and/or inspect features orportions of the model 124 in the new location with respect to the realphysical environment. In some aspects, the AR system 100 may enable themodel 124 to be moved continuously in real time as the user operates theuser interface 160, as an alternative to inputting a movement commandconsisting of X, Y, and Z coordinate movement amounts. The AR system 100may also enable the user to alter the configuration of the model 124 ofthe conveyor system. For example, the user may provide inputs to theuser interface 160 to reconfigure, add, or remove individual modules orwork stations within the conveyor system represented by the model 124.

In some aspects, whether or not the user operates the AR system 100 tomove the model 124 to different locations within the real physicalenvironment or re-configure the conveyor system of the model 124, theuser may perform a detailed inspection, noting clearances and/orinterferences between the model 124 and features of the real physicalenvironment. The user may physically measure (e.g., with a tape measure,ruler, etc.) such clearance or interference distances. For example, theuser can measure a distance from the model 124 to one or more of thefixed utility features 152A to 152E or to other features or equipmentsuch as other work stations, storage facilities, handling equipment,railings, walkways, etc. The relative positions and/or orientationsbetween the model 124 of the conveyor system and other models or realphysical environment features can also be inspected while the model 124is displayed in the real physical environment. When relating features ofthe real physical environment to features of the model 124, the ARsystem 100 may enable the user to virtually identify, select, or targetcertain features of the real physical environment and/or features of themodel 124 by applying “tags”. Applying a tag can include selecting thelocation of the desired feature(s) with a pointer or by moving acrosshair for example. Once tagged, the processor 108 may automaticallycalculate a spacing or interference distance measurement, which can bedisplayed and/or saved to the memory 104 for later reference. With tagsapplied, the AR system 100 may further enable the user to reposition themodel 124, or reconfigure at least a portion of the model 124, accordingto a user defined value for a minimum or maximum proximity from one ormore of the building features such as the fixed utility features 152A to152E.

As shown in FIG. 6, the user interface 160 can include various appletsfor the user to interact with the AR system 100. Across the top of thedisplay, a first set of applets control the display size of hotspots(e.g., with a slider control), the showing of guides (e.g., on/off), andthe showing of the model 124 (e.g., on/off), respectively. Hotspots canrefer to tagged areas of the model 124 that may provide access tofurther information and/or functionality within the AR system 100. Inaccordance with the above description regarding measurements, anotherapplet is provided to measure distances. Further, additional applets areprovided for positioning, scaling, and rotating the displayed model 124.The positioning applet can include a speed slider control, one or moregroups of position arrows that move the displayed model 124 alongperpendicular axes in accordance with the selected speed, and a pair ofdirection buttons that adjust the proximity, i.e., bringing the model124 closer or moving it farther away. The scale applet can be providedin the form of a slider control that increases or decreases the modelscale. The rotation applet can include a speed slider control, aturntable toggle (e.g., on/off), one or more rotation arrows, and a setof turntable direction buttons (e.g., clockwise/counterclockwise). Whenthe turntable toggle is off, the model 124 is displayed in singlelocation and does not move unless adjusted, for example, by rotatingabout perpendicular pitch and yaw axes with the rotation arrows. This isseparate and additional to the position controls, which translate thedisplayed model 124. Finally, when the turntable toggle is on, the model124 will rotate clockwise or counterclockwise so that the user(s) do nothave to move around the model 124 to obtain different perspectives. Theturntable mode may rotate the model 124 continuously, and the speed ofrotation may be set by the speed slider control. The turntable directionbuttons change the direction of rotation in turntable mode.

In some aspects, either in combination with operations described aboveor separately, the user may put the conveyor system of the model 124into virtual operation. For example, the user may interact with themodel 124 directly (e.g., virtually operating controls of the controlstation 134) or indirectly through the user interface 160, according toany of the embodiments or command types discussed above. Accordingly,operating the conveyor system of the model 124 can include any one ormore of: starting conveying, stopping conveying, moving work pieces 140into and out of enclosures (e.g., furnace, coating booth, drying kiln)of the model 124, changing a speed of conveyance, loading work pieces140, unloading work pieces 140, changing among various sizes/shapes ofwork pieces 140. Operating the conveyor system of the model 124 canoptionally further include simulating a process by which one or morework pieces 140 are dipped into an immersion tank and lifted therefrom(e.g., electrocoating or at autodeposition bath). Such a virtualoperation of the model 124 can also include tilting and/or rotating theone or more work pieces 140 to one or more alternate orientations. Theaxis or axes of movement can be predefined features of the conveyorsystem, or may be selectable by the user during interaction with themodel 124 while using the AR system 100. In one particular method, theuser can command starting of conveyance of one or more work pieces 140,and the user can physically walk along with the work piece 140 as it isconveyed through a work station or a series of work stations defining aprocess (e.g., finishing, cleaning, coating, painting, etc.). The ARsystem 100 can enable the user to identify a position and a path of workpieces 140 with respect to the real physical environment (e.g., any ofthe fixed utility features 152A to 152E) as the work pieces 140 movealong the virtual conveyor system. The AR system 100 may further beoperable to make adjustments to the characteristics of the model 124during the interaction with the display device 116. As non-limitingexamples, any of the following may be adjusted by the user: conveyorheight, work piece size or shape, line density of work pieces along theconveyor, and work station (e.g., oven, tank, booth) height, length,and/or width.

FIGS. 2-5 illustrate another type of conveyor system that can bedisplayed and interacted with via the AR system 100. The conveyor systemcan take the form of that disclosed in U.S. Pat. No. 9,468,944. Thisadditional embodiment provides perspective on additional ways ofoperating the AR system 100, but by no means is the invention limited tothe exact types of conveyor systems shown in the drawings. The conveyorsystem of FIGS. 2-5 includes three parallel conveyor tracks 336A to 336Cthat support a work piece 340 by suspending three points of the workpiece 340 from the respective conveyor tracks 336A to 336C. The workpiece 340 in this embodiment is a liquid permeable container or basketthat may be used to carry a plurality of smaller manufactured items forprocessing (e.g., cleaning, finish application by autodeposition ore-coating). Reference number 324 indicates the model created by thedisplay device 116 of the AR system 100 from a three-dimensional modelof the conveyor system. One or more of the conveyor tracks 336A to 336Cvirtually embodied by the model 324 can be individually movable forvertical travel relative to the others to impart various tiltingoperations of the work piece 340. The tilting operations can includetilting laterally left and right as shown in FIGS. 2 and 3, or tiltingfore and aft as shown in FIGS. 4 and 5 (directions identified relativeto the direction of extension of the conveyor tracks 336A to 336C andlikewise the conveyed travel of the work piece 340).

Methods of operating the AR system 100 with the model 324 can includeany or all of the steps described above with respect to the model 124,including measuring, moving, reconfiguring, and controlling virtualoperation. Measuring, for example, can include first virtually operatingthe conveyor system model 324 into a particular state of operation, andthen measuring between a tagged portion of the model 324 and a featureof the real physical building in which the model 324 is displayed.Virtual operation of the conveyor system model 324 can includemanipulating individual ones of the conveyor tracks 336A to 336C withrespect to the others to impart a tilting of the work piece 340. Becausethe position and orientation of the work piece 340 changes with respectto one or more of the conveyor tracks 336A to 336C during suchoperations, it may be particularly beneficial for the user to be able tomake observations, inspection, and/or measurements based on a particularphase or state of the virtual operation of the conveyor system.

In some embodiments, the AR system 100 enables two or more individuals,each having a respective display device 116, to operate the AR system100 simultaneously. In such embodiments, the display devices 116 cancommunicate with respective processors 108 and/or memory devices 104, orcan share (e.g., communicate with, receive and/or transmit information,etc.) a common processor 108 and/or memory device 116. Also, in suchembodiments, the users can have the same or different types of displaydevices 116, and can operate in any desired respective locations in thereal environment. The ability of multiple users with respective displaydevices operating in the same environment enables users to cooperate inviewing, moving, or modifying the model as desired, and in understandingaspects and features of the model.

Some embodiments of the AR system 100 can also enable one or more usersto operate the system 100 using VR display devices (not shown), in whichthe such display devices do not necessarily enable the VR display deviceusers to see the real environment in addition to the model informationbeing displayed as described above. However, even in cases where one ormore users can only see, move about, manipulate and/or control the modeldata on their VR displays (i.e., without seeing the surrounding realenvironment), this capability still enables users to join in anoperating session of the AR system 100 with AR display device userswhile being remote from the real-world location. In such cases, theremote users using VR display devices can participate using respectivedisplay devices, transceivers, and memory devices in communication withone or more other processors 108 of the local AR participants, such asover the internet or any computer network.

In some embodiments, the information viewed by a user of the AR system100 (i.e., through any of the AR display devices 116 described above)can also be simultaneously viewed on one or more other display devices.In such cases, the model being displayed on the AR display device 116 isalso transmitted to and simultaneously displayed upon the other displaydevice(s), which can be local or remote with respect to the real-worldenvironment in which the model is being displayed, such as over anysuitable network or over the internet. In such cases, the real-worldenvironment viewed by the AR display device user can also be captured,transmitted, and simultaneously viewed by another user on any local orremote display device over the same connection(s) by one or more camerason the AR display device 116. Such cameras can simultaneously capturevideo of the real world environment viewed by the AR display device inwhich the model is being displayed. In this manner, another local orremote user viewing any display device (e.g., a computer screen, AR orVR display device, and the like) can see what one of the AR displaysystem users is seeing (i.e., the model and the real-world environment)through that user's AR display device 116, and can do so in real-time ornear real-time over one or more computer networks and/or the internet.This capability enables individuals to observe what the AR system useris seeing, doing, and experiencing in the AR environment.

What is claimed is:
 1. An augmented reality system comprising: a displaydevice defining a display area; a database containing athree-dimensional model of a conveyor system; a communication linkbetween the database and the display device; a processor operable todisplay the three-dimensional model of the conveyor system on thedisplay area, wherein the display allows a human user to see a real timecombined view of the real physical environment, including a building andfeatures thereof, and the three-dimensional model of the factoryautomation system within the building; and a user interface havingcontrols operable to effect a movement of the displayed conveyor systemwithin the building and controls that are operable to tag a feature ofthe displayed conveyor system and perform a measurement between thetagged feature and a feature physically present in the building.
 2. Theaugmented reality system of claim 1, wherein the display device includeseyewear, and the display area is transparent or semi-transparent.
 3. Theaugmented reality system of claim 1, wherein the electroniccommunication link is established through a wireless network.
 4. Theaugmented reality system of claim 1, wherein the user interface has atleast one control operable to effect starting of virtual operation ofthe displayed conveyor system.
 5. The augmented reality system of claim1, wherein the user interface is provided on the display device.
 6. Theaugmented reality system of claim 5 wherein the display device includeseyewear.
 7. The augmented reality system of claim 1, wherein the displaydevice is a handheld personal electronic device.
 8. A method ofoperating an augmented reality system in a building, the methodcomprising: providing a display device to a human user in the building,the display device defining a display area; displaying athree-dimensional model of a conveyor system to the human user, whilesimultaneously allowing the human user to see real physical features ofthe building in real time to give the impression that the conveyorsystem is installed in the building, wherein displaying thethree-dimensional model of the conveyor system includes presenting aconveyor system control panel; starting virtual operation of thedisplayed conveyor system in response to a command from the user via aninput to a user interface; and tagging a feature of the displayedconveyor system and measuring between the tagged feature and a featurephysically present in the building.
 9. The method of claim 8, whereinthe user interface is a remote physical device separate from the displaydevice.
 10. The method of claim 8, wherein the user interface isprovided on the display device.
 11. The method of claim 8, wherein theinput is a gesture or voice command.
 12. The method of claim 8, furthercomprising changing at least one parameter of the virtual operation ofthe displayed conveyor system in response to a further input to the userinterface, the at least one parameter being selected from a groupconsisting of: changing a work piece handled by the conveyor system,changing a position of a work piece relative to a conveyor track of theconveyor system, changing an orientation of a work piece relative to aconveyor track of the conveyor system, changing a speed of conveyance,or changing a path of conveyance.
 13. A method of operating an augmentedreality system in a building, the method comprising: providing a displaydevice to a human user in the building, the display device defining adisplay area; displaying a three-dimensional model of a conveyor systemto the human user, while simultaneously allowing the human user to seereal physical features of the building in real time to give theimpression that the conveyor system is installed in the building;tagging a feature of the three-dimensional model of the conveyor systemwith a user interface; performing with the user interface a measurementbetween the tagged feature of the three-dimensional model and at leastone utility feature of the building; and performing with the userinterface a movement of at least a portion of the displayedthree-dimensional model to a new position within the building.
 14. Themethod of claim 13, further comprising starting and stopping a virtualoperation of the displayed conveyor system in response to one or moreinputs from the human user to the user interface.
 15. The method ofclaim 13, further comprising manipulating one or more parameters of thedisplayed conveyor system with a gesture or voice command.
 16. Themethod of claim 13, further comprising calculating with a processor aclearance or interference between the tagged feature of thethree-dimensional model and the at least one utility physical feature ofthe building.
 17. The method of claim 13, wherein the display device isprovided as eyewear, and the three-dimensional model of the conveyorsystem is displayed on a transparent or semi-transparent portion of thedisplay area to provide a direct view of the real physical features ofthe building.
 18. The method of claim 13, wherein the three-dimensionalmodel of the conveyor system is displayed on the display area along witha computer-generated view of the real physical features of the building.